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The Legislative Audit Committee 
of the Montana State Legislature: 


This is our information systems audit of the Tracking Remedial and Environmental 
Actions Data System (TREADS) managed by the Waste Management and 
Remediation Division at the Department of Environmental Quality. 


This report provides the legislature information about system development and project 
management of the new system. This report includes recommendations for creating 
and enforcing a system development framework with defined roles and responsibilities, 
improving technical risk mitigation, and improving communication and status 


reporting. A written response from the department is included at the end of the report. 
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The Tracking Remedial Environmental Actions Data System (TREADS) iscurrently 
in its third year of development. The Department of Environmental Quality has 
spent roughly 63 percent of the project’s current budget of $5.3 million, but in 
2016 the department terminated its relationship with the original development 
contractor and moved to in-house development. The department faces challenges 
with system development processes and communication, specifically engagement 
and collaboration. The project currently lacks a defined project framework in 
which roles and responsibilities of the project team are clear. The department also 
has limited assurance that system development contains organized, deliberative, 
and cost-effective methodologies. While the department recently made efforts to 
improve TREADS development, additional steps are necessary to mitigate project 


REPORT SUMMARY 


risks to ensure system expectations are met. 


Context 


The Department of Environmental Quality’s 
(department) Waste Management and 
Remediation Division (division) investigates 
and oversees cleanup of environmentally 
contaminated sites. It oversees environmental 
protections related to solid waste and 
underground storage tanks, as well as issuance of 
permits and licenses. It prepares the appropriate 
environmental review documents to comply 
with the Montana Environmental Policy Act. 
‘The division is currently using legacy systems 
to manage its processes. Although these are 
currently functioning as needed, the various 
programs within the division do not have a 
centralized system for sharing, maintaining, 
and reporting environmental and financial 
data. 


Due to the antiquated legacy systems, the 
department initiated the Remediation 
Information Management System project 


in 2012 and was approved for Long-Range 
Information Technology House Bill (HB) 10 
funding during the 2013 Legislative Session 
for $1.8 million, which includes $700,000 
from the General Fund, $1 million from 
a state special revenue fund, and $40,000 
from federal funds. The Tracking Remedial 
Environmental Actions Data System 
(TREADS) includes common management of 
site and client data, full project data, activity 
support for participating programs, and 
integrated workflow management, document 
management, sample data management, 
and mapping capabilities. The new system 
will enable the department to better align 
with business processes and provide a more 
efficient approach to sharing, maintaining, 
and reporting data through a centralized 
system. The project spans six programs and 
one administratively attached board within 
the division and is currently expected to be 
implemented in June 2018. 


(continued on back) 
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The department is currently developing the 
custom system internally after the contract 
ended with the original design, develop, and 
implement (DDI) third party vendor. Since the 
department is spending considerable internal 
resources on the development of TREADS, 
this audit reviewed the system development 
and project management processes to help 
ensure TREADS will meet expectations of 
the system and end users. Specific areas of 
review included mitigation of business and 
technical risks and overall communication 
effectiveness and transparency. Conducting 
the audit while the system is being developed 
allows the department to make improvements 
to the process through the remainder of the 
project. 


Results 


Based on the audit work performed we 
determined the department is missing a 
clear and defined project framework. Audit 
work also found the department lacks 
assurance that system development contains 
organized, deliberative, and cost-effective 
methodologies. Without a framework based 
on best practices, team members cannot 
follow or adhere to project requirements, 
objectives, and goals. We found limited 
engagement and collaboration. TREADS 
team members had little understanding of 
the documented framework or their roles 
and responsibilities within TREADS. By 
defining and enforcing a project framework 
and roles and responsibilities, the department 
will have assurance the system development 
project is efficient and effective. 


Although the department addressed 
business risk and applied specific design 
complications within development, we 
identified key technical risks were not being 
mitigated through sprint development. 
Although progress is being tracked at a high 
level, improvements can be made in overall 
progress reporting and communication of 
status to both internal and governmental 
stakeholders. Audit work also identified 
a lack of healthy culture necessary for a 
successful project. Overall, we developed six 
recommendations to address audit findings. 


Recommendation Concurrence 


Source: Agency audit response included in 
final report. 


For a complete copy of the report (17DP-01) or for further information, contact the 
Legislative Audit Division at 406-444-3122; e-mail to lad@mt.gov; or check the web site at 


http://leg.mt.gov/audit 
Report Fraud, Waste, and Abuse to the Legislative Auditor's FRAUD HOTLINE 


Call toll-free 1-800-222-4446, or e-mail lad@mt.gov. 


Chapter | — Introduction and Background 


Introduction 


The Department of Environmental Quality’s (department) Waste Management and 
Remediation Division (division) investigates and oversees cleanup of environmentally 
contaminated sites. It oversees environmental protections related to solid waste and 
underground storage tanks, as well as issuance of permits and licenses. It prepares 
the appropriate environmental review documents to comply with the Montana 
Environmental Policy Act. This work includes coordination and preparation of 
environmental impact statements, ensuring methods and standards are consistent with 
department policy, and coordination with regulatory programs in the division, the 
department, and other state and federal agencies. The division conducts all facility 
inspections and reviews reports to determine compliance with permit conditions, laws, 


and regulations. 


Division’s Legacy Systems Need 
Replacing to Improve Reporting 


The division is currently using a legacy database to manage its processes. Although 
it is currently functioning as needed, the various programs within the division do 
not have a centralized system for sharing, maintaining, and reporting data. The 
department currently faces challenges with providing consistent and accurate public 
information reports to stakeholders. Data reported from the department can include 
environmental waste cleanup data, environmental samples, and financial data. Since 
the legacy database is not a viable enterprise solution, the division is in need of a more 
adaptable system that meets the needs of all programs within the division. Therefore, 
the department is working toward the development of a new information system. ‘This 
system as a whole is referred to as TREADS (Tracking Remedial and Environmental 
Actions Data System). The overall replacement project is called the Remediation 
Information Management System (RIMS). It includes data management and sample 
data management solutions, which were included in the evaluation of development 
work completed by department. 


New System Will Manage Data for 
Various Departmental Programs 


TREADS will be a single point data entry (one system access point for data entry) 
web-based system that will support the division. The multiple programs the system 
will serve include Underground Storage Tank Section, Petroleum Tank Cleanup 
Section, Cleanup Protection Redevelopment Section, State Superfund Unit, Federal 
Superfund and Construction Services Bureau, Abandoned Mine Lands Section, and 
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the administratively attached Petroleum Tank Release Compensation Board. The 
system will manage data, environmental samples, and documentation as well as have 
an integrated spatial data system. TREADS is expected to align with the following 
objectives and system goals: 


¢ Adaptable 

¢ Modular 

¢ Easy to Maintain 

¢ Easy to Use 

¢ Seamlessly Integrated 
* Secure 


¢ Improve Business Processes 


Funding and Timeline for a New System 


The department was approved for a new management information system in the 
2013 Legislative Session through the Joint Appropriations Subcommittee on Long-Range 
Planning, It received $1.8 million in House Bill 10 funding. In 2014, the overall 
budget totaled $4.7 million with fund sources including General Fund, state special 
revenue, and federal special revenue. An implementation date of December 2016 was 
set. After funding was received, the department began searching for a vendor through 
the request-for-proposal process. A year later, the department awarded the contract to 
a design, develop, and implement (DDI) vendor and requested supplementary funding 
in the appropriation of $820,000 for additional functionality for the Underground 
Storage Tank Section and the Petroleum Tank Release Compensation Board. 


The project kicked-off in the fall of 2014. In June 2015, the department reported to the 
Legislative Finance Committee (LFC) that the TREADS project had fallen behind 
due to DDI vendor scheduling issues; however, by September 2015 the project was 
reported as being back on track. 


By March 2016, the department had spent 47 percent of its budget, but began 
encountering performance and personnel issues with the DDI vendor. The vendor 
believed it was meeting contractual deliverables and objectives. However, the 
department disagreed and eventually the contract was terminated in June 2016. The 
department decided to move the project to internal development. It was reported 
to LFC in June 2016 the updated budget added an additional $500,000, totaling 
$4.8 million with a new delivery date of December 2017. Finally, in May 2017, the 
budget increased to $5.3 million with a new delivery date of June 2018. Furthermore, 
additional funding was acquired for temporary development staff to assist in TREADS 
development. The department conducted a transition analysis and shifted resources to 


internal development and implementation, where it continues to develop a custom 
system. The following table summarizes the funding and timeline changes made 
throughout the project up to this point. 


Table 4 
TREADS Budget and Implementation Date Overview 


Estimated Budget Proposed System Reason for Increase 
(millions) Implementation Date 


March2oig_ | 88 | Notyetset_ 


Added EE for Petroleum 
Tank Release Compensation 

Mareneny $2.62 Mongols Board and the Underground 
Storage Tank Section 


September 2014 $4.27 June 2016 DEQ staff time added (soft costs) 


Refined cost estimate of 
June 2016 $4.77 December 2017 department and contracted staff 
time 
Soft cost increase due to 
Pecemieatenle $5.34 areas department taking over project 
Source: Compiled by the Legislative Audit Division using Legislative Finance Committee and 
department records. 


Project Management and System 
Development Methodology Selected 


Project management is the application of knowledge, skills, tools, and techniques to 
anticipate activities to meet the expected project requirements. Project management 
is accomplished through the appropriate application and integration of project 
management processes, which are categorized into the following process groups: 

¢ Initiating: This process group includes the basic groundwork necessary to 


create the project and define the guidelines and criteria under which it will 
operate. 


¢ Planning: This process group establishes stakeholder expectations and 
makes clear how the project will be managed. 


¢ Executing: This phase is where the project’s technical work takes place. 


¢ Monitoring and Controlling: This process group ensures that project 
deliverables are on time, on budget, and of acceptable quality. 


¢ Implementing: This process is the final stage of the project where the system 
is implemented into a live environment, final details are submitted, and 
obligations are completed. 
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A development framework for an information system serves to organize, program, 
and supervise the process of developing an information system. There are different 
frameworks for information systems development. Development framework is 
composed of the process groups deployed in the development or investment of a 
specific software system. The framework must be used to structure, plan, and control 
the process. Many frameworks exist; however, there are two primary development 
frameworks that are recognized as being the most common. 


1. Waterfall framework, which is seen as the traditional method. 


2. Agile framework, which is seen as the newest type of framework used in 
system development. 


While each framework has advantages and disadvantages, each one can result in 
successful system development. 


Department Selects an Agile/Scrum Framework 


Agile relates to a method of project management, used especially for software 
development. There are different development approaches to an agile framework, 
including kanban, extreme programming (XP), and scrum. Agile/scrum is a specific 
approach within the agile family. The original DDI vendor followed an agile/scrum 
framework. After that relationship ended, the department decided to continue using 
agile/scrum as the primary framework. The department had used an agile/scrum 
approach for smaller projects in the past; however, it had never used this framework on 
a large-scale system until TREADS. 


An agile/scrum approach is a simplistic incremental framework that follows all process 
groups over a two- to three-week period. In other words, one piece of the system goes 
through Initiating, Planning, Executing, Monitoring/Controlling, and Implementing 
during a short period of time known as “sprints.” 


Within an agile framework, the project phases are intended to be flexible and adaptable. 
Agile best practices emphasize the following core values: 


¢ — Individuals and interactions over processes and tools. 
¢ Working products over comprehensive documentation. 
¢ Customer collaboration over contract negotiations. 


¢ Responding to change over following a plan. 


The agile/scrum model is the most common form of agile development. According 
to best practices, the product owner (key business stakeholder) creates prioritized 
high-level requirements called a product backlog. Work is completed in iterations or 


sprints, and each sprint is two to three weeks long with a piece of working product at 
the end of each sprint. Within these sprints are user stories. “User stories” are specific 
tasks that are assigned to the project team members. User stories are assigned story 
points, which are typically measured by the amount of time and effort it will take to 
complete the task. Once the task is completed and if time permits within the sprint, a 
team member moves to the next task. During sprint planning, the team pulls a small 
chunk from the product backlog and decides how to implement those pieces. Each day 
during the sprint, the team holds a daily standup meeting (daily scrum meeting) or a 
brief meeting to assess progress and retrospectives. 


Another key player within an agile/scrum environment is the scrum master. This 
person keeps the team focused on the goal. At the end of the sprint, work completed 
should be ready to use by end users. Burndown/up charts are used to help measure 
ideal and remaining work (efforts) for the sprint. To present the new functionality, a 
sprint review and retrospective meeting occurs (product owner demonstration). The 
cycle begins again until a functioning system is implemented. When agile/scrum is 
operated effectively, it ensures stakeholders receive regular deliveries of a product they 
can use. The effectiveness of the agile/scrum approach largely depends on the team 
and the collaboration efforts between team members. The following figure is a visual 
representation of the agile/scrum framework. 


Figure 1 
Agile/Scrum Framework Process 
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Audit Scope 


This audit focused primarily on the department’s current design and development of 
TREADS and examined the department’s system development process, not the actual 
system. Therefore, this audit does not provide assurance over the specific system that 
will be implemented. Additionally, fieldwork began shortly after the relationship ended 
with the original DDI vendor and concluded at the end of May 2017, for a total of six 
months. ‘Therefore, unless specifically noted, the audit focused on the department's 
overall internal system development practices and internal communications during 
that point in time. We also looked at what and how the department was reporting 
project status to Department of Administration’s (DOA) State Information Technology 
Services Division (SITSD) and the Legislative Finance Committee (LFC). 


This audit excludes the role of DOA SITSD oversight of projects as well as its review of 
reported costs and budget due to Information System and Performance Audits currently 
examining Governance over IT Spending and State Government Procurement and 
Contract Management. 


Audit Objectives 


Adopting a specific framework to organize and implement a new system is important 
to ensure the design, development, and testing of the system is completed accurately 
and efficiently. Audit planning identified risk in the department’s alignment with 
agile/scrum best practices and overall transparency surrounding the system. The three 
objectives developed were: 

1. Determine whether Department of Environmental Quality’s development 


methodology aligns with best practices to ensure an effective system 
development process. 


2. Determine ifthe Department of Environmental Quality is mitigating business 
and technical risk by implementing controls during sprint development. 


3. Determine status of TREADS and how status is communicated and reported 
to internal and governmental stakeholders. 


Audit Methodologies 


Steps taken to conclude on our objectives included: 


¢ Developed an assessment matrix using best practices to determine and rate 
the effectiveness in the department’s project management framework. 


¢ Researched and compiled best practices and resources to apply in the 
evaluation of TREADS. 


¢ Compiled, analyzed, reviewed seven TREADS teleases, 21 sprints, and 
associated user stories to determine overall system health. 


¢ = Interviewed 29 TREADS stakeholders, including product owners, subject 
matter experts, steering committee, technical team, and external government 
officials. 


¢ Observed staff, processes, and interactions to assist in comparison to best 
practices and evaluation of system development. 


¢ — Identified, tested, reviewed, and evaluated the department’s documentation 
and collaboration applications. 


¢ Observed various meetings including project status meetings, sprint planning 
meetings, product owner demonstrations, and steering committee meetings. 


Overall Conclusion 


Based on the audit work performed, we determined the department is missing a clear 
and defined project development framework for TREADS. Although the department 
addressed business risk and specific design challenges within sprint development, 
key technical risks were not being mitigated through sprint development. Although 
progress is being tracked at a basic level, improvements can be made in overall 
progress reporting and communication of progress to both internal and governmental 
stakeholders. We also determined improvements to the TREADS project work culture 
could increase the success of the project. 


Since this is an audit of system development, the audit team communicated audit 
findings on an ongoing basis throughout the audit. Therefore, the department has 
begun taking steps to address audit recommendations discussed throughout the report. 


Report Contents 


‘The remainder of this report includes additional background and details of our findings, 
conclusions, and recommendations and is organized in the following manner: 
¢ Chapter II addresses overall alignment with agile/scrum best practices, 


development industry standards, effectiveness and efficiency of current 
. Y . eJe ie . . r Y 
practices, and roles and responsibilities within the project. 


¢ Chapter III explains and discusses the department's mitigation of risk 
through sprint development, specifically in the area of technical risk. 


¢ Chapter IV presents information regarding the department’s communication 
and project status reporting. 


Chapter II — Significant Improvements 
Needed to Align With TREADS 
Development Methodology 


Introduction 


The first objective of our audit was to determine whether the Department of 
Environmental Quality’s (department) development methodology aligns with best 
practices to ensure an effective system development process. This chapter discusses 
details regarding agile/scrum best practices. It addresses needed improvements to 
ensure the Tracking Remedial and Environmental Actions Data System (TREADS) 
will strengthen department business processes and be delivered on time and within 
budget. We also discuss our comparison of team members’ roles and responsibilities 
within the department to best practices. This chapter also addresses potential project 
risks associated with internal development practices and the improvements the 
department will need to make to ensure internal resources are organized and effective 
for project development. 


Prior to fieldwork beginning, the department underwent a reorganization and 
experienced turnover in several key staff positions. TREADS was currently underway 
when the reorganization occurred, which resulted in several staff in the project changing 
positions. Furthermore, during the reorganization the department also terminated its 
relationship with the design, develop, and implement (DDI) vendor and made the 
decision to move the project to internal resources and development. 


Department of Environmental Quality Reorganization 


In May 2015, the department began an “optimization project” to analyze opportunities 
for increasing the effectiveness of the department and its services. The guiding principles 
of organization optimization included: improve effectiveness, provide for resource 
capacity and flexibility, promote a culture of continuous improvement, foster employee 
and program innovation, improve stakeholder and customer services, integrate related 
work units and enhance communication, fully use staff expertise across the agency, 
and create ability to focus on outcomes without sacrificing the integrity of processes. 


The department analyzed where programs intersect, how internal support services 
function across the organization, and what opportunities exist for programs to work 
more closely together in a more effective pursuit of the department’s mission. The 
result of this work was a major reorganization of several department functions. The 
reorganization grouped programs with similar functions (air, land, water), stakeholders, 
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and skill sets. The effective date of the reorganization was July 2015; however, pieces 
were still moving until March 2016, which was also the time the department began 
facing performance and personnel issues with the DDI vendor. Additionally, staff 
throughout Central Services Division and the Waste Management and Remediation 
Division shifted supervisors and managers. For example, one project manager of 
TREADS originally reported to the division administrator, but was moved and began 
reporting to the bureau chief for Contaminated Site Cleanup Bureau. Other changes 
specific to the TREADS project included: 
¢ Remediation Division changed to Waste Management and Remediation 
Division. 
¢ Waste Management and Remediation Division now includes Waste and 
Underground Tank Management Bureau. 


¢ Office of Information Technology moved under Centralized Services 
Division. 


The following figures (see pages 11 and 12) show the areas related to TREADS before 
and after the department-wide reorganization. 


Figure 2 
Organizational Structure Prior to Reorganization of Department 
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Figure 3 


Organizational Structure After Reorganization of Department 
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Department Conducted Transition 
Analysis for Internal Development 


After ending the contract with the original DDI vendor in July 2016 due to a 
disagreement in achieved deliverables, the department conducted a transition analysis 
to evaluate the state of the project and identify any corrective action plans needed 
to complete the project internally. The document provided a list of alternatives to 
move forward. ‘The project’s steering committee, which oversees the project, chose an 
alternative that included new system requirements and a system rebaseline. System 
or project requirements are statements that outline needed system capabilities and 
expectations. System rebaselining includes making significant database table changes 
or changes to its original structure. The changes also included workflow refactor 
(reorganization of the automation of data flow) and table consolidations to provide 
a new foundation for the system. The goal of the agreed-upon alternative allows 
workflow process to be more flexible and efficient. Workflows within a system allow 
steps of specific processes to be automated. ‘The transition analysis predicted these 
changes would take an additional six sprints or twelve weeks. During fieldwork, it 
was discovered the department was still in the process of rebaselining the system six 
months after the rebaselining process began. The transition analysis also documented 
the department’s use of agile/scrum framework for the project’s official framework as 
well as the team members roles and responsibilities in the project. 


Hybrid Approach to Agile/Scrum Established as Framework 


Although project documentation outlines an agile/scrum approach to TREADS 
system development, the department emphasizes it is using a hybrid approach of agile/ 
scrum. It believes this is the best approach for the development of TREADS. A hybrid 
approach is a mix between agile and waterfall. According to best practices, although 
a hybrid approach has been known to work in some cases, the core values of an agile 
approach must be present. For example, a successful approach to a hybrid method 
may include development of requirements but keeps the demand for constant delivery 
within a limited time frame, or iteration. Additionally, as long as the development team 
maintains good communication, effective cooperation, and a coordinated approach, a 
hybrid methodology may yield successful results. 


Section 2-17-505, MCA, and industry standards require project development must 
be effective, efficient, organized, and deliberate. The framework chosen must fit 
and be appropriate for the size and objectives of the project. Agile/scrum does not 
follow a prescribed framework found in project management methodologies, such as 
waterfall, rather it is a value driven framework that focuses heavily on collaboration 
and functionality over documentation and contract negotiation like other frameworks. 
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Although agile/scrum is flexible in nature, not establishing a defined, industry 
recommended framework can significantly increase risks within a project, such as 


missing important system requirements or extending budgets or timelines. 


Agile Values Reviewed, Compared to Department’s 
Overall System Development Process 


Agile best practices emphasize the values surrounding individuals and interactions, 
working products, customer collaboration, and responding to change quickly. 
Through interviews with the TREADS team members, they indicated that a strict 
agile approach would not benefit the project. However, after we explained some of the 
values of an agile approach, it was clear the agile principles and values were not fully 
understood. Although the team members knew the basics of an agile approach, many 
staff members had limited knowledge and understanding of agile values, principles, and 
best practices. Furthermore, according to the department, the team received training in 
agile/scrum; however, since the break with the original DDI vendor, refresher training 
for the team did not occur. Although staff received initial agile/scrum training, they 
were not as entrenched in the development of TREADS due to the involvement of the 
DDI vendor, whereas now the project has moved to internal development, staff are 
required to have a more in-depth knowledge of the framework. 


Department’s Hybrid Approach Does Not Meet Agile Values 


Through observations of meetings and general interactions as well as interviews with 
all TREADS team members, it was apparent the department had not fully considered 
how a hybrid approach would best fit the department’s needs considering what agile 
best practices recommends for a successful project, specifically with taking a hybrid 
approach. The department’s use of a hybrid approach includes added use of waterfall 
elements, such as defined project or system requirements, and development of 
documentation. We reviewed the use of these elements and found them being used 
appropriately; however, where the department falls short in meeting hybrid approach 
best practices is continuous product backlog management (reviewing and prioritizing 
work tasks during each sprint’s planning discussions), collaboration, interaction, and 
producing a working product within each iterative cycle. 


When we reviewed the product backlog, we discovered that the backlog had not been 
revisited since the break with the original DDI vendor. Backlog grooming is vital in 
an agile development framework and consists of the product owners and the project 
team reviewing items to ensure the project still contains appropriate items and the 
most important pieces of the system are prioritized. Activities that can occur during 
backlog grooming include removing user stories that are no longer relevant, creating 


new user stories based on new needs, and correcting user story time estimates based 


on new information. For the TREADS project, backlog grooming becomes even more 
important since multiple product owners with differing priorities are involved. 


Additionally, while the department uses defined project or system requirements, 
we found those requirements were not consistently being tied back to user stories 
and current functionality. Tying project requirements to user stories ensures that 
expectations are being met through sprint development. Due to the lack of product 
backlog grooming and tying project requirements to user stories, there is an unclear 
picture of what has been completed and what still needs to be accomplished. 


We also determined that a working product was not produced at the end of each 
sprint. The current work completed showed that basic system functionality was not 
being produced and testing was not occurring on a consistent basis. After reviewing 
five releases or fifteen sprints, we found that roughly 48 percent of stories completed 
were focused on administrative tasks such as planning and meetings, whereas industry 
standards advise less focus on administrative tasks and more focus on producing a 
working product at the end of each sprint in order to meet customer needs. 


During fieldwork, we addressed the various findings with the department. It was 
reiterated that because of the previously hired DDI vendor, the department did not 
want to go strictly agile. The department believed the hybrid approach effectively 
addressed the concerns previously identified with the vendor, including personnel and 
performance issues. 


During the audit, the department began addressing the missing agile values and 
principles. This includes tying project requirements back to user stories and obtaining 
training on agile work/time estimations. It also hired a summer intern to assist in 
quality assurance and testing practices and produced a working product in the latest 
release of the system. However, without a clear framework, project management cannot 
be streamlined. Additionally, because the department is missing a defined framework, 


its expectations and enforcement are unclear. 


Risk Increases Without Defined System 
Development Framework 


Without a defined system development framework and without staff being 
knowledgeable and trained on the specific framework, project risk increases. The 
department has fallen behind schedule and it risks the objectives and goals of the system 
not being met on schedule and budget. An absence of defined development framework 
also provides unclear progress for system development and the team does not understand 
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or have a clear direction of the status of the project. Figure 4 summarizes responses 
from TREADS team members during interviews. We interviewed 27 TREADS team 
members and many had multiple concerns. 


Figure 4 
Overview of TREADS Team Members Interview Responses 
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Source: Compiled by the Legislative Audit Division. 


Additionally, without a clear, defined, and accepted framework, the department risks 
undetected bugs, overall system issues, and potentially unusable functionality, which 
could possibly lead to user workarounds, contradicting the overall project objectives. 
An undefined framework can also provide misunderstood or missing project 
requirements and project objectives, which can result in users not getting a system they 
budgeted for and expected. Furthermore, without trained and knowledgeable staff on 
the defined framework, efforts lack coordination. Other effects of not having a defined 
and established framework with trained team members include: 


¢ Unable to integrate with the department’s strategic IT plan, information 
architecture, and technology direction. 


¢ Costly reworking. 


¢ — System security and confidentiality compromised. 
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RECOMMENDATION #1 


We recommend the Department of Environmental Quality: 


A. — Establish and clearly define the development framework to be used in 
TREADS development. 


B. Ensure all staff with a role in TREADS development are trained on the 
established development framework. 


TL 


Agile Roles in TREADS Reviewed and Assessed 


Agile best practices emphasize the values surrounding individuals and interactions, 
working products, customer collaboration, and responding to change quickly. Industry 
leaders also outline roles and responsibilities within an agile framework. An example 
is a product owner, or owner of the system, who is typically one person. Other roles 
include development team and scrum master, whose primary responsibility is to keep 
the project team on track. During the spring of 2017, we observed and interviewed 
the TREADS team members. We observed interactions, conversations, and overall 
engagement in the project. While gaining a better understanding of team members’ 
roles and responsibilities within an agile framework, we compared the department's 
practices to industry best practices. Furthermore, we reviewed the overall approach to 
agile roles and responsibilities. 


Although the department held required meetings and created tasks for team members 
using user stories, we observed that defined roles and responsibilities were not enforced 
or understood. In addition, through interviews with TREADS team members that 
included bureau chiefs, section supervisors, and technical staff, we discovered there 
was a lack of understanding of the various roles and responsibilities within the project, 
including their own. It is important for all TREADS team members, regardless of 
position in the department, to have a clear understanding of the agile values and 
principles for an organized approach to development and implementation. It also 
provides clear roles within the project to ensure deliberative work is accomplished. 


Importance of Roles and Responsibilities 
in System Development 


Roles, responsibilities, and accountability can be found throughout many project 
management industry standards. Although policy does not exist for project 
management framework, the State Information Technology Services Division (SITSD) 
encourages the adherence to the Project Management Body of Knowledge (PMBOK). 
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PMBOK is a collection of processes, best practices, terminologies, and guidelines 
that are accepted as standards within the project management industry. According to 
PMBOK, human resource planning is used to describe how roles and responsibilities, 
reporting relationships, and staffing management will be addressed and structured 
within a project. Effective human resource planning should consider and plan for the 
availability of or competition for scarce resources. Other projects may be competing 
for resources with the same skill sets, which could affect project costs, schedules, risks, 
quality, and other project areas. 


Roles and Responsibility Documentation 
Lacks Enforcement 


Department documentation provides several outlines of the TREADS team roles and 
responsibilities. The documentation initially provided clear direction of the expectations 
from each team member. However, staff turnover occurred and many team members 
either do not understand their roles and responsibilities or have other conflicting 
department work tasks that do not allow them to put the time and resources into their 
TREADS project responsibilities. 


Through observations and interviews, many of the TREADS team members were 
unclear about their roles and responsibilities within the project. It was found that a 
staffing plan or human resource plan had not been maintained or enforced for the 
project team. Although the department had roles and responsibilities documented, 
these roles and responsibilities had not been updated consistently when staff turnover 
occurred and were not being enforced as outlined by industry standards. 


Confusion Among TREADS Team Members 
Led to Non-Prioritized Work Completed 


Staff were unclear about their role and responsibility within TREADS project because 
the department reorganized, but the uncertainty can also contribute to the loss of the 
DDI vendor, and the department taking over the project internally. Team members 
performed work that was supposed to be completed by other staff. For example, the 
project manager, without communicating to the team, completed user stories or 
delegated tasks detailing specific pieces of the system, which were originally assigned 
to technical staff. Although best practice indicates team members are encouraged to 
take stories from the backlog, regardless of who was assigned, staff become unclear on 
what is expected of them in a sprint without proper communication. Furthermore, 
by not communicating that tasks were completed by different staff, development of 
a specific feature risks not working cohesively with rest of system. Another example 
includes inconsistent testing conducted without input from product owners. In a 
typical waterfall approach, different levels of testing occur by the subject matter expert 


at each level. However, in an agile framework, testing is done by all team members at 
the same time. This allows for consistent transfer of information and clear expectations. 
Additionally, audit work found that temporary contracted support were performing a 
majority of the work. By allowing this to happen, the department risks losing key 
information as well as internal staff relying heavily on temporary staff to complete 
their tasks. Without a clear, updated, and enforced staffing plan, the communication 
of knowledge risks being lost and expectations missed. 


Additionally, work tasks were completed that were not vetted through the steering 
committee or included as part of the transition analysis. Because there was not a clear 
understanding or enforcement of team members’ roles and responsibilities, work was 
completed without project management knowledge or approval. This increased the 
number of tasks in the backlog, which then caused additional work for the contracted 
authorized temporary development staff, inevitably costing the department additional 
resources to extend the temporary development staff contract. 


The department did not have enforcement and accountability for the system 
development team, because there were not well-defined roles and responsibilities for 
each team member. ‘There were also communication and coordination issues because 
the reporting structure was not clearly defined. These issues reduced the collaboration 
between team members, which resulted in the TREADS project not being developed 
as efficiently as it should. This issue became more elevated once the development team 
began to experience staff turnover and a department reorganization changed staff 
roles within the department. A formalized project-staffing plan defining the roles and 
responsibilities of system development staff and a required reporting structure could 
have improved the overall effectiveness of the system development team. 


Me 


RECOMMENDATION #2 


We recommend the Department of Environmental Quality develop and 
enforce a TREADS project staffing plan that: 


A. Clarifies and defines roles and responsibilities of all team members. 


B. Establishes reporting and communication structure. 
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Chapter III - Managing TREADS Project Risks 


Introduction 


The second objective of our audit was to determine whether the Department of 
Environmental Quality (department) is mitigating risk by implementing controls 
during sprint development. Sprint development is a set period of time during which 
specific work has to be completed and made ready for review. This chapter discusses 
details regarding risk mitigation and how the department applied risk mitigation 
through sprint development. Technical risk deals with unproven assumptions in 
emerging design that might significantly affect the ability to deliver the solution. 
Technical risk also deals with feasibility and means there is a proven solution that can 
be delivered within the time and cost constraints of the project. Technical risk includes 
security of the system, system performance, and automation of data integrity. Specific 
technical risk examples include user interface stability, data integrity controls, and 
network connectivity with other systems. By applying controls to address these specific 
technical risks, assurance is given that the new system is secure, accurate, and efficient. 


Mitigating System Risk in User Stories 
and Product Backlog Is Important 


The department took steps to address business risks; however, audit work found 
that technical risk had not been addressed or mitigated. Application of controls and 
monitoring of such risks includes applying specific language within a risk register 
(official log of both business and technical risks associated within the project), product 
backlog (maintained project requirements), during sprint development. 


We compiled various best practices and standards from industry leaders in agile/scrum, 
specifically in mitigating risk in the selected framework. Best practices recommend 
applying technical risk through user stories and the project’s product backlog. It also 
highly recommends applying and addressing risk early and often, with a transparent 
and open dialogue. 


Mitigating technical risk is important because if these risks are not addressed, the 
final system risks the possibility of security breaches, broken functionality, or limited 
performance expectations. Addressing these risks improves the chances of minimal 
issues after implementation. Additionally, the earlier the risks are addressed, there is 
more assurance the system will be implemented on time and on budget. 


21 


22 


Montana Legislative Audit Division 


TREADS Technical Risk Mitigation Reviewed 


Based on reviews, while the original design, develop, and implement (DDI) vendor 
was still contracted and developing TREADS, it made a consistent effort to address 
and mitigate technical risks during sprints and releases. During previous development 
efforts with the contracted vendor, the department, with vendor guidance, applied 
technical risk mitigation to sprint development. This included user stories assigned 
to security, segregation of duties, or network risks as well as screenshots of developed 
functionality to address technical risk such as role-based assignments and the path 
for assigning roles. The DDI vendor appeared to follow agile/scrum best practices 
and attempted to include end users (product owners, subject matter experts) into the 
development of processes and features. However, the department ended the contract in 
2016 and ended the practices previously conducted by the vendor, including mitigating 
risk during sprint development. 


Through review of user stories, sprints, and other documentation, we found that 
technical risk considerations had not been revisited or applied to the backlog since 
the break with the vendor. Additionally, while many technical risks surfaced from 
the shift to internal development and were discussed in various meetings, these risks 
had not been added to the project’s official risk register. Additionally, user stories are 
missing evidence that these considerations have been applied to sprint development 
or the product backlog. Furthermore, documentation regarding technical and 
security considerations has not been updated to reflect the break with the vendor or 
considerations on technical risks associated with limited code from the vendor. 


The department has also not begun developing systematic reports such as error reports, 
audit of system changes, and data checks. This is scheduled for later on in the project 
but user stories or potential system problems or issues have not yet been created to 
address these system requirements in the product backlog. Therefore, the potential 
exists for these risks to be overlooked as the project reaches budget and timeline 
constraints. 


Issues Faced if Technical Risk Unmitigated 


Since the department did not have consistent and transparent technical risk mitigation 
through user stories and sprint development, problems with TREADS functionality 
and integration occurred. For example, the department's transition analysis estimated 
an additional twelve weeks of work, but the work is at least six months behind 
schedule due to unintended technical complications such as integration of previous 
DDI source code and internal customized coding of new table and workflow 
structure. If technical risk is not mitigated, the department may create unpredicted 
functional and technical issues that can affect final implementation. There is also a 


higher risk of missing important system controls such as access privileges and data 
checks. Additionally, without consistently and proactively considering technical risk 
and ensuring documentation is updated, the department risks encountering additional 
technical issues that may further delay the project. 


The lack of transparency associated with technical risks gives management a less 
comprehensive picture of project status. While the department believes the project is 
notat the stage to begin addressing technical risks, best practices and industry standards 
indicate technical risk should be addressed throughout the project’s life cycle. We 
would expect to see technical risk being addressed in the project’s risk register, product 
backlog, and during sprint development, so these considerations not only remain on 
the department's radar, but also provide a road map in case there is staff turnover. 


Although the department began meeting to address to the various technical risks, 
overall the department is missing consistent and transparent risk mitigation during 
sprint development. In order to mitigate risk most effectively, business and technical 
tisk should be addressed early and often in the system development process. 
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RECOMMENDATION #3 


We recommend the Department of Environmental Quality create consistent 
and transparent technical risk mitigation for TREADS by: 


A. — Adding technical risk mitigation to the project tasks and risk register. 


B. Prioritizing technical risk mitigation during development. 
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Chapter IV — Internal and External 
Communication of the TREADS Project 


Introduction 


The third objective of our audit was to determine the status of the Tracking Remedial 
Environmental Actions Data System (TREADS) and how status is communicated 
and reported to internal and governmental stakeholders. This chapter discusses details 
regarding communications and how the Department of Environmental Quality 
(department) needs to make improvements to better ensure information it reports 
on the TREADS project is open and transparent. This chapter also discusses the 
department's status reporting and how status is reported to executive management, 
the Department of Administration’s (DOA) State Information Technology Services 
Division (SITSD), and the Legislative Finance Committee (LFC). 


Communication Key to Project Management 


and System Development 


The Project Management Body of Knowledge (PMBOK) outlines several standards 
for all communication within a system development project. PMBOK is the entire 
collection of processes, best practices, terminologies, and guidelines that are accepted 
as standards within the project management industry. Although communication can 
come in many forms and avenues, effective and collaborative communication practices 
are required for any project to be successful. According to agile values, collaboration, 
individuals, customers, and interactions are vital to a successful project. All of these 
elements can be considered part of the larger communication structure within the 
project. 


TREADS Spans Multiple Department Bureaus 


Communication becomes even more important when a system spans across multiple 
internal stakeholders, which is the case with TREADS. The new system will support 
six programs and one administratively attached board within the Waste Management 
and Remediation Division (division). The goal of the system is to be able to share and 
maintain data in one centralized location for the division. Although data from each 
section is used for different purposes, the data will be able to be shared to serve the 
different purposes within the division; each program will have its own process template 
and customized workflow process for various purposes such as electronic submittal of 
forms and task lists. 


The programs included in the project are: 
¢ Underground Storage Tank Section 


¢ — Petroleum Tank Cleanup Section 
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¢ Cleanup Protection Redevelopment Section 

¢ — State Superfund Unit 
Site Response Section 
® — State Superfund Section 

¢ Federal Superfund and Construction Services Bureau 
Abandoned Mine Lands Section 


¢ Petroleum Tank Release Compensation Board 


It is also worth noting that although the Petroleum Tank Release Compensation Board 
(PTRCB) does not report directly to the division, it reports to an external board. 
PTRCB has additional financial and reporting requirements that will be specific to 
the board. Each program has a variety of staff members with different interests in the 
system and each has its own product owner and/or subject matter expert. Although 
requirements have been set for the project, it is important to have a decision-making 
body to prioritize not only at the bureau level but at the agency level, and to ensure 
requirements are met as well as resolve any conflicts that may arise. Due to the 
division-wide goals of being able to share data, communication between the seven 
programs is imperative to ensure needs are met, but yet a flexible enough system to 
serve as a division-wide tool. The following section discusses specific areas reviewed 
and improvements that can be made in the department’s internal communication and 


overall project success. 


Project Participation and Meeting 
Attendance Needs Improvement 


Through observations of meetings and staff interactions, we determined that 
participation and collaboration were limited. Many meetings occurred during the 
observed 15 sprints. For each sprint, we tracked attendance for the product owner 
demonstration, project status, and sprint planning meetings. We grouped the 30 team 
members into five areas of expertise, including Project Managers (2 team members); 
IT Operations (5 team members); Subject Matter Experts (11 team members); Product 
Owners (7 team members); and Executive Management (5 team members). If a team 
member was excused from the meeting, we noted them as attended. Best practices call 
for full participation and attendance from all team members in order to meet effective 
collaboration and communication values. Therefore, we would anticipate 100 percent 
attendance from each group. As Figure 5 (see page 27) shows, only the project status 
meetings had 100 percent attendance by Project Managers, Subject Matter Experts, 
and Executive Management, whereas IT Operations and Product Owners only 
attended 27 percent and 93 percent, respectively. 


Figure 5 
TREADS Participation and Attendance Analysis 


Sprint Planning Meeting Attendance 


Project Manager _ IT Operations Subject Matter Product Owners Executive 
Experts Management 


Project Status Meeting Attendance 


Project Manager __IT Operations Subject Matter Product Owners Executive 
Experts Management 


Product Owner Demonstration Meeting Attendance 


Project Manager _IT Operations Subject Matter Product Owners Executive 
Experts Management 


Source: Compiled by the Legislative Audit Division 
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As illustrated in the figure, many product owners did not actively participate in 
meetings and conversations. As part of our audit work, we reviewed and evaluated 
several sprints occurring over many months and we found attendance fell behind 
approximately halfway through the sprints we evaluated. Agile/scrum best practices 
and industry standards recommend consistent inclusion and participation in all project 
related meetings. This provides a collaborative and communicative environment in 
which successful projects can be attributed. Additionally, this environment provides an 
opportunity for project members to become cross-trained. 


Our observations identified several other concerns: 


¢ Meetings were often short and with limited communication and did not 
accomplish items that are expected in an agile framework. 


¢ Team members were ill-mannered and disrespectful to each other during 
day-to-day conversations, and some meetings regarding TREADS 
conversations were lacking in professional and respectful communication. 


¢ Key project staff often lacked engagement and interest in the project. 


Due to the agile/scrum framework, professional and civil communication and 
collaboration are necessities. Projects with successful and effective communication 
practices have more transparency and the ability to correct issues in an efficient 
manner. This leads to better system implementation because all team members will 
be focused on meeting the needs and expectations of a final product. As can be seen 
by the participation in the meetings above, as well as our staff interviews and meeting 
observations, these pieces of agile values were not always present. 


Department Not Following Project Communication Plan 


The department created a communication plan at the beginning of the project, 
which has been updated once since ending the contract with the design, develop, and 
implement (DDI) vendor. The document outlines policies and procedures on internal 
and external communications regarding the project, however after reviewing the 
documentation and through meeting observations, we determined the department 
was not following their developed procedures. 


The communication plan outlines review and approval of meeting minutes and 
documents, however we found these were not approved or in many cases not reviewed 
by necessary staff before the next meeting. Many times, topics previously discussed 
were brought up again because staff had either not attended the previous meeting and 
notes were not reviewed, or follow-up to the documents/discussions had not occurred. 


Another example in the communication plan where the department was falling short 
included updating the Requirements Traceability Matrix (RTM) and the product 
backlog. The RTM is the initial set of system expectations or project requirements. 
These listed project requirements were used in the overall project decision package 
and outline all required pieces of the new system. The product backlog serves a similar 
purpose, however the product backlog is designed for an agile approach. Due to the 
hybrid approach the department is taking to agile, using both pieces is important 
in tracking progress and projections. It was discovered that the RTM was not being 
updated consistently or accurately based on current work, nor was the product backlog 
updated and maintained. Therefore, information communicated during meetings was 
not always transparent. 


Although the department discussed implementing deliverables or milestones outlined 
in the communication plan, there is limited evidence this actually occurred despite the 
procedural requirement outlined in documentation. While the department discussed 
implementing these elements in future releases, milestones and deliverables have yet to 
be delivered. The shortage of clear priorities and set milestones/budgets also effects how 
much commitment is given to the project from TREADS team members. ‘The effects 
of missing a collaborative and interactive culture/environment have led to disengaged 
TREADS team members, disorganization, and an overall absence of professional 


communication within the project. 


Multiple Factors Impact Project Success, 
Making Formal Communication Plan Vital 


While various changes attributed to confusion surrounding the project, including major 
department reorganization, loss of key staff, and severing ties with the original DDI 
vendor, ultimately the project team has not reestablished its communication plan since 
these events. This communication breakdown causes unclear priorities and input from 
management, limited management knowledge, and lack of understanding of project 
and project framework. This in turn can potentially cause inaccurate information to 
be shared with stakeholders, resulting in resources to not be allocated appropriately to 
meet users’ needs, schedule, and budget. 


The department does not have effective and transparent internal communication 
practices. Because it does not follow or update its communication plan, staff and team 
members do not have a full understanding of overall department communications and 
team members do not fully understand the hybrid approach to agile/scrum values. 
These issues are occurring because the department’s communication plan has not been 
clearly aligned with TREADS development framework. 
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RECOMMENDATION #4 


We recommend the Department of Environmental Quality update and align its 
communication plan with its TREADS development framework. 


TO 


Overall Communication and Steering Committee 
Engagement Needs Improvement 


The department’s project communication plan also describes the purpose of the 
project’s steering committee. The steering committee serves as the final decision point 
for the project and is composed of product owners, the division administrator, and key 
technical staff such as the technical lead, project managers, chief information officer 


(CIO), and the chief security officer (CSO). 


Engagement and Prioritization of TREADS 
Varied Among Project Team 


During fieldwork, we determined the TREADS steering committee did not meet on 
a consistent and ongoing basis from October 2016 to February 2017. Additionally, 
there was miscommunication and misdirection between the TREADS team, steering 
committee, and management. Many times management reiterated the project was a 
high priority, but when speaking with subject matter experts and technical staff, those 
priorities were not always enforced. Many team members had conflicting work that 


took priority. 


Additionally, based on interviews with product owners and TREADS team members, 
as well as numerous observations of project-related meetings, we identified several 
challenges in internal communications. It was clear the challenges with rebaselining 
(changes to system’s foundation), even though they were discussed, were not fully 
understood by TREADS team members and executive management. We also identified 
miscommunication regarding the overall prioritization of the project and how product 
owners and the steering committee communicated priorities to the TREADS team. 


It is shown through the risk register, the log of all project risks, that staffing resources 
are a risk. However, there is still a misrepresentation of priorities between management 
and project staff. Meaning the team keeps moving forward with current time 
estimates; however, team members work on other departmental priorities, impacting 


the project. This can be shown through evidence of user stories moved/deleted in a 


sprint, attendance and participation, and daily communications. The breakdown in 
communication caused delays in schedule and overall transparency. There is a lack of a 
clear picture of what remains to be done in system development and what is complete. 
Without a clear picture, internal stakeholders do not have a thorough understanding of 
project health or have the ability to set department-wide priorities. 


Agile Values Focus on Communication and Collaboration 


In a traditional agile approach, there is typically only one product owner. However, 
because there are multiple product owners in this specific project, it is even more 
important the steering committee meets to ensure the project is on track, on budget, 
and that risks are mitigated. Additionally, because several programs have invested in 
the project, a decision-making group is required to determine priorities of the project 
at the agency level. 


Overall Improvements to Culture Need to 
Occur to Ensure Effective New System 


During the early part of our audit fieldwork, we identified technical difficulties with 
TREADS source code, personnel issues, competing legislative session priorities, 
and overall absence of engagement among staff. Once these findings were discussed 
with the department, the steering committee did begin holding regular twice-a- 
month meetings. However, through continuing observations of various meetings 
and interviews with staff, we observed staff members were still not effectively 
communicating with each other throughout the project, including steering committee 
meetings. While small project changes have occurred and were approved by the 
steering committee, the impediments and roadblocks faced by the project are left 
largely undiscussed by the committee due to limited collaboration and interaction 
between team members. Overall, the TREADS project has been adversely affected 
by an unhealthy organizational culture, which needs to be addressed directly at the 
departmental level. In particular, due to the existence of several product owners, it is 
important to have a decision-making body to prioritize the system’s needs and goals, so 
the department needs to focus its efforts in the role of the steering committee. 


RECOMMENDATION #5 


We recommend the Department of Environmental Quality establish, prioritize, 
and enforce a project culture of collaboration and interaction by ensuring 
TREADS steering committee takes an active, engaged, and consistent role in 
the project. 


TO 
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Following Best Practices Improves Governmental 
Reporting and Communication 


Various laws and rules require certain information on state IT projects be reported. 
All business application systems funded through House Bill 10 in the 2013 Legislative 
Session must have had a plan approved by the state of Montana’s CIO to determine if the 
implementation plan fits according to the department and state’s needs. Additionally, 
rules require agencies to report progress of software and management. Agencies that 
receive House Bill 10 funding over $500,000 are required to report to the Legislative 
Finance Committee (LFC) per committee rules. Agencies are required to report status 
on costs, budgets, and overall system health through SITSD where the information is 
compiled and presented to LFC. Due to high value systems and agencies receiving tax 
dollars for the systems, LFC reviews and assesses risks of those systems. This not only 
holds the agency accountable for spending tax dollars, but also provides oversight of 
new information management systems developed throughout the state of Montana. 


The department currently reports to the LFC as required. Due to the higher risk in 
developing a custom system as well as time and budget concerns, the CIO, and LFC 
have listed TREADS as medium risk over various stages of the project. The department 
reports it is 65 percent done with the project. The project managers clarified that the 
65 percent refers to the design and development phase of the project, not the project 
as a whole. The department maintained the June 2018 implementation date, but 
decided to break up the implementation into two production releases. The department 
has yet to determine what these two releases will look like. Because of the length of 
time this system development project has currently taken, the number of changes 
in the schedule, and the overall high risk in developing custom systems, ensuring 
communication is clear and accurate while considering the documented agile/scrum 
framework is required. 


Agile Best Practices Use Velocity 
Calculations and Burndown Charts 


The department currently uses Gantt Charts (a chart in which a series of horizontal 
lines shows the amount of work done or completed in certain periods of time in relation 
to the amount planned) to provide a visual of progress and projection. However, these 
kinds of visual aids do not provide an accurate picture of progress in this type of project 
management framework. Velocity calculations are part of agile/scrum best practice tools 
and are readily available through the department’s use of an issue tracking application. 
Within the application, multiple calculations exist to provide a way to communicate 
the overall health of the project. The calculations are done to measure the speed at 
which work is delivered in a sprint. The velocity rate is calculated to determine how 


much value has been delivered thus far, when all user stories in the product backlog 
will be delivered, and how many story points will be delivered by a certain date. Story 
points are an assigned number based on the time and effort it will take to complete a 
task. The velocity calculations then produce a velocity chart. Because the department 
did not develop this calculation, we created a rough calculation based on current 
completed story points per each sprint. Our best-case scenario calculation provided an 
implementation date several months past the department's projected implementation 
date of June 2018. This timeline was also validated by two external contractors hired 
by the department as part of the project. 


Another effective estimation and planning tool used in an agile/scrum environment, 
which is readily available in the issue tracking application, is the burndown chart. 
Burndown charts display the remaining effort for a given period, usually within a sprint 
or release. The department uses burndown charts for sprints; however, we determined 
the department is not estimating story points and efforts effectively, therefore producing 
inaccurate calculations and not meeting deadlines for produced functionality. Figure 6 
(see page 34) provides an example of Sprint 52 through Sprint 54 burndown charts. 
Although the example is a small sample of reviewed sprints, the burndown charts 
consistently did not meet industry best practices. During fieldwork, the department 
attended an estimation training to better align the charts to best practices, however 
we found improvements were not being made. The gray line represents the amount 
of work within the sprint (ideal effort), which is measured by the total story points 
assigned in a sprint. The red line shows the remaining work (effort) during the sprint. 
The shaded gray represents the non working days. 
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Figure 6 
Example of Department’s Burndown Charts for Sprint 52 through Sprint 54 


Sprint 52 
(March 2,2017 - March 16, 2017) 


Story Points for Sprint 52 


Mar 16 
Mar 15 


Sprint 53 
(March 16, 2017 - March 30, 2017) 
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Sprint 54 
(March 30, 2017 - April 14, 2017) 


Story Points for Sprint 54 


Mar 30 


Source: Compiled by the Legislative Audit Division using department records. 


While the department uses burndown charts, an effective and proper agile/scrum 
framework should reflect best practices, as shown in the figure below. The best practices 
figure shows the effort remaining in the sprint and the ideal effort hours closely 
aligning. If efforts and calculations had been managed effectively, the department's 
burndown charts would better reflect best practices. 


Figure 7 
Example of Best Practice Burndown Chart 
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Source: Scrum Alliance, scrumalliance.org. 


As discussed above, the department was not tracking progress and projections 
accurately. It uses a project management and issue tracking application, which provides 
several agile tools to assist in providing velocity calculations and various progress 
reports such as burndown charts and monitoring and control reports. Audit work 
found that these readily available tools to assist in schedule and budget calculations 
were not being used. This included the use of velocity charts, proper use of burndown 
charts, and overall estimation tools for the entire project. The department was aware 
the project was missing the usage of the readily available agile tools; however, staff 
lacked the necessary resources and knowledge of these tools. The department has since 
provided staff training on project budget and estimations in an agile project and is 
actively trying to improve overall estimations of story points and user stories. 


Effects of Steering Committee Disengagement 
on External Communications 


We found that milestones and deliverables had not been created, and communication 
on progress and projections was limited. This has been discussed several times during 
steering committee meetings; however, these calculations had not been delivered 
as of the end of audit fieldwork. The department tracks its progress and projections 
through Gantt Charts, but does not use a work rate ratio for calculation. Although 
Gantt Charts are effective tools for project management, when following an agile/ 
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scrum approach, there are better, more accurate calculation tools that can be used in 
an agile framework. Furthermore, the department’s TREADS Gantt Charts provides 
inaccurate estimations and has limited transparency due to constantly moving 
implementation dates and undeveloped functionality. 


The absence of consistently scheduled steering committee meetings created a lack 
of updated and current knowledge being communicated to executive management, 
eventually resulting in missing key information being communicated to governmental 
stakeholders. For example, when we met with Legislative Fiscal Division (LFD) staff 
it was discovered that the LFC was not aware that rebaselining was still occurring. 
Additionally, when interviewing the state CIO we discovered SITSD was not aware 
of the agile/scrum hybrid approach the department was taking or the turnover in key 
staff. We also determined meetings were not documented with the details needed to 
make a clear assessment of project issues with TREADS. Although the department 
aligns with required statutory reporting, improvements in transparency could be 
made if calculation tools and steering committee reporting occurred consistently and 
accurately. 


Finally, the department decided to break the implementation into two production 
releases. As audit work continued, we found that functionality and other features of 
the system are being pushed further behind. Because of staff turnover and conflicting 
system needs between previous and existing staff, other programs with high investment 
dollars have yet to begin development. Since there will now be two production releases, 
it is also unclear if all project objectives and goals will be implemented according to the 
release date provided to LFC. 


Improvements to Project Budget and 
Timeline Estimates Need to Occur 


Although the department adheres to reporting processes and schedules required by 
SITSD and LFC, improvements can be made to ensure governmental and internal 
stakeholders understand the current progress the department is making with 
TREADS. Furthermore, the department should be using readily available tools to 
assist with estimating and communicating project status. Although the department 
uses Gantt Charts to provide visuals of progress and scheduled timeline, due to the 
nature of an agile/scrum project, providing constant and updated information based 
on current sprints, story points, and releases will better reflect timelines and progress. 


Me 


RECOMMENDATION #6 


We recommend the Department of Environmental Quality: 


A. Develop calculation tools that reflect best practices for TREADS 
established framework. 


B. Use developed tools to track time, establish deliverables and milestones, 
and project TREADS schedule and budget. 
C. Use projections to better communicate TREADS status to internal and 


governmental stakeholders. 
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November 1, 2017 RECEIVED 


Mr. Angus Maciver NOV O1 2017 
Legislative Auditor AUDIT DIV. 
Legislative Audit Division LEGISLATIVE 

PO Box 201705 


Helena, MT 59620-1705 
RE: TREADS Information Technology Audit #17DP-01 
Dear Mr. Maciver, 


Thank you for the opportunity to respond to the Information Technology Audit #17DP-01 for the 
Department of Environmental Quality. We have reviewed the recommendations contained in the 
report and provided our responses below. 


RECOMMENDATION #1 

We recommend the Department of Environmental Quality: 

A.) Establish and clearly define the development framework to be used in TREADS 
development. 

B.) Ensure all staff with a role in TREADS development are trained on the established 
development framework. 


Response #1 

A.) DEQ concurs with the Legislative Audit’s Division’s recommendation to have a clearly 
defined project framework. Throughout the project, we have defined and attempted to use a 
hybrid approach on the project. The goal was that software development was to follow an 
agile software development approach and general project management will follow best 
practices as prescribed in the Project Management Body of Knowledge. This has had some 
success, but has led to some confusion on roles and responsibilities and enforcement of the 
framework. By December 1, 2017, DEQ will update its documentation to provide more 
clarity where and how best practices are used for general management of the project and 
clearly define the development framework to be used. 

B.) Once DEQ revises its framework, all staff with a role in TREADS will attend required 
training on the established project and software development framework. 


RECOMMENDATION #2 

We recommend the Department of Environmental Quality develop and enforces TREADS 
project staffing plan that: 

A.) Clarifies and defines roles and responsibilities of all team members 

B.) Established reporting and communication structure. 
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Response #2 

A.) DEQ concurs with the Legislative Audit’s Division’s recommendations. There has been staff 
turnover and transition in the development of TREADS that have led to challenges. DEQ has 
made steps to address these concerns and we will be continuing with efforts to clearly define 
roles and responsibilities of all team members. Once DEQ revises its project management 
and software development framework (as discussed in Recommendation 1), we will evaluate 
roles and responsibilities of all team members and revise these where necessary. The 
estimated timeframe for this is by mid-November 2017. 

B.) DEQ will ensure that there is a reporting and communication structure for project decision 
making, escalation of issues and concerns, and fully embracing a Steering Committee 
approach. The estimated timeframe for this is by mid-November 2017. 


RECOMMENDATION #3 

We recommend the Department of Environmental Quality create consistent and transparent 
technical risk mitigation for TREADS by: 

A.) Adding technical risk management to the project tasks and risk register 

B.) Prioritizing technical risk mitigation during development. 


Response #3 

A.) DEQ concurs and has informed all team members that they are responsible for identifying 
technical risk as part of the project’s risk management process. DEQ has been identifying 
technical risks since the beginning of the project and is working towards mitigating those risks. 
B.) DEQ will incorporate additional guidance in our communication strategy for communicating 
and prioritizing those risks. 


RECOMMENDATION #4 
We recommend the Department of Environmental Quality update and align its communication 
plan with its TREADS Development framework. 


Response #4 
DEQ concurs with the Legislative Audit’s Division’s recommendations as stated in our response 
to recommendation #2. 


RECOMMENDATION #5 

We recommend the Department of Environmental Quality establish, prioritize, and enforce a 
project culture of collaboration and interaction by ensuring TREADS steering committee takes 
an active, engaged, and consistent role in the project. 


Response #5 

DEQ concurs with the Legislative Audit’s Division’s recommendations. DEQ will re-evaluate 
the structure, roles, responsibilities, and communication tools/process of the Steering Committee 
to better support a project culture of collaboration and interaction. 


RECOMMENDATION #6 

We recommend the Department of Environmental Quality: 

A.) Develop calculation tools that reflect best practices for TREADS established framework. 

B.) Use developed tools to track time, establish deliverables, and project TREADS schedule and 
budget. 

C.) Use projections to better communicate TREADS status to internal and governmental 
stakeholders. 


Response # 6 

DEQ concurs with the Legislative Audit’s Division’s recommendations. 

A.) DEQ will use and in some cases, continue to use calculation tools that best reflect the 
TREADS Project Management and Agile Software Development framework. For example, at 
the end of a two week sprint, a written status report will be submitted to the Steering 
Committee and Product Owners for review. Each report will include the following 
information: 

. Summary of status for scope, budget, and schedule 
b. Summary of issues and risks impacting the scope, budget, or schedule, including 
project staffing and technical risks 

Work accomplished during the current iteration 

Work planned for the next iteration 

Status of delivery for each major deliverable in the project 

Sprint Burndown chart which shows the trend in completing project tasks 

Velocity chart to help determine team velocity and help with estimating what team 

can achieve in future sprints 

h. Updated Deliverables and Milestone chart showing the current status of the project. 

B.) DEQ will use developed tools to track time, establish deliverables, and project TREADS 
schedule and budget. 

C.) DEQ will use projections to better communicate TREADS status to internal and 
governmental stakeholders. With that said, DEQ is using the Legislative Finance 
Committees’ established reporting requirements for state projects which are based on Earned 
Value Management. Due to DEQ’s awareness of the heightened risk associated with the 
project, DEQ provided a supplemental report during the last LFC reporting period. 
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I want to thank you and your staff for the professionalism and fairness during the audit fieldwork 
and conferences. We appreciate the willingness of the auditors to discuss recommendations and 
respond to our questions. We always look upon the audit process as an opportunity to improve 
the department’s operations and performance. 


Sincerely, : 
— 7 : 
7 L Pet rf toe 
Tom Livers 
Director 


Department of Environmental Quality 


